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ABSTRACT 



Strategies and systems for evaluating educational 
programs arc detailed in an attempt to fulfill the need for a 
centralized evaluation capability. Since a fundamental prerequisite 
for program evaluation is a statement of the objectives of the 
educational system, a set of system goals is proposea. These goals 
would provi e a framework of criteria for analyzing educational 
programs and for determining policies and procedures. An evaluation 
data base is then suggested as a means of handling the accumulation, 
storage, processing, and analysis of the vast amount of information 
necessary vo the evaluation process. The development, and capabilities 
of such an evaluation data system are outlined. The Planning, 

Research and Evaluation Division is considered as basic to the 
implementation of the evaluation strategy. The division's proposed 
functions would include the establishment of definitions, measures, 
criteria, and planning goals for programs in relation to system 
objectives; the maintenance of a record of all special and 
experimental programs and the evaluation of the same; the development 
of a student, staff, program, and school data system for use in both 
short and long term evaluation; and finally, the direction of 
research studies focused on the improvement of the entire evaluation 
process. Appendices include models which implement the data system 
for analysis, guidelines for design and analysis strategies, 
suggestions for project monitoring, description of studs n t-or iented 
data processing, and relevant references. (PR) 
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SUMMARY 



A. Overview 

In this report, strategies and mechanisms for educational eval- 
uation are presented. The point of reference for evaluation discussed 
here is the Division of Planning, Research and Evaluation. The primary 
assumption is that there is a need for a centralized evaluation cap- 
ability, and that this capability cculd provide useful services even to 
decentralized or locally controlled school districts. 

The greatest service an evaluation function in a public insti- 
tution can perform in the long run is to make information available. At 
present in the D.C. Public Schools there is an enormous amount of infor- 
mation in the form of records, reports, computer printouts and so on. 
For purposes of evaluation, however, many existing data are functionally 
non-existant. They must be assembled by laborious searches of records 
and analyzed in elementary ways owing to a lack of machinery for proces- 
sing and analyzing evaluative data. 

Since programs are what a school system offers its students, 
the focus of evaluation should be programs, and the measures of effec- 



tiveness of programs the extent to which they benefit the system's 
constituents i.e., students. But programs do not exist without a con- 
text. They are means by which the educational system carries out its 
mission, the vehicles for accomplishing objectives. An important pre- 
requisite, then, for evaluating programs, is a statement of the objectives 
of the educational system - a statement of the results that programs are 
intended to achieve, and criteria for measuring accomplishment of them. 

A set of system objectives is suggested. They are broad cate- 
gories of conditions to which different policies, procedures and programs 
may be addressed. The categories of objectives serve as a framework for 
analyzing the multitude of programs initiated by the school system, and 
as references for evaluation of programs in the short run as well as in 
the long run. Means of achieving major system objectives are not stated 
or recommended, as this is not within the scope of evaluation. Deter- 
mining means or approaches is part of the planning function, and because 
of this there needs to be a close linkage between planning and evalu- 
ation if evaluation is to serve a practical or constructive purpose. 

A function of evaluation at the system level should be to provide contin- 
uous or periodic feedback to planners and decision-makers on the extent 
to which various programs, arrangements or approaches seems to be con- 
tributing to the accomplishment of system objectives, and to provide recom- 
mendations for Improvements or changes as part of the feedback. 

Statements of objectives lead to determinations of criteria, 
the standards by which accomplishment of objectives may be measured. 
Criteria for each major category of objective are suggested. Once 
criteria are established, the requirements for measurement and data can 
be defined, and from these definitions mechanisms or systems for data 
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collection, analysis, and reporting can be established. 

Major objectives of the school system, no matter how defined, 
are concerned with the educational achievement and development of students, 
and with the policies, processes and programs that help define and bring 
about such achievement. In order to assist in focussing on how programs 
and processes are in fact contributing to educational development of 
students under varying conditions, information that relates students to 
programs is needed. The information required should enable analysis to 
be made of the comparative effects of different types of educational 
inputs (programs, services, etc.) with students of different educational 
needs. To handle the accumulation, storage, processing and analysis of 
such information, an evaluation data base is proposed. The data base is 
intended to maintain an historical record of students as they proceed 
through the educational system. It is not intended to be a general 
management information system, or pupil or program accounting system. 

It is a tool for evaluation of program effects on a gross level. The 
data base, which would contain data collected periodically, would be 
supplemented by soecial field studies aimed at more extensive measurement 
and description of educational process variables and conditions of 
instruction. The data base would also provide a means of identifying 
groups participating in special projects or programs (e.g., Title III 
projects) for followup studies of the long-term effects of such projects. 

Measures of the functions of programs are obtainable partly 
from performance measures to be stored in the data base, and partly from 
analyses of the relationships of special or Innovative projects to system 
objectives and operations. 
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B. Proposed Functions of the Planning, Research and Evaluation Division^ 

The Planning, Research and Evaluation Divison should work with 
other divisions and departments, and with other groups in the school 
system and community, to establish definitions, measures, criteria and 
planning goals for programs in relation to system objectives. 

The Division should be responsible for assuring that the impacts 
of special projects and programs are projected, and that changes in pro- 
jections owing to changes in programs or other conditions are made and 
disseminated appropriately. 

The Division should be responsible for maintaining a record of 
all special and experimental projects and programs in the system and for 
assisting in the planning and/or implementation of evaluation of them. 

It should help to assure compatabi li ty of measures and instrumentation, 
and to assure that appropriate evaluative information is available for 
further system planning and decision-making at all levels. 

The Division should start the development of student-staff- 
program-school data system that will enable both short term and longitu- 
dinal evaluations of programs and their benefits to students. The 
Division must be able to coordinate with other departments and groups in 
and out of the educational system in Identifying programs to be evaluated, 
criteria to be employed, and instruments to be used. It must be able to 
incorporate a variety of Information in the data base, perform a number 
of general and special analyses, and report Independently its findings 
and recommendations. The data base that is developed must be capable of 

1. The official title of the Division Is still the Division of 
Planning, Innovation and Research. However, it Is referred to through- 
out the report as the Division of Planning, Research and Evaluation, at 
the request of the Acting Division Head, Dr. Mildred P. Cooper, to be 
consistent with the emerging functions and organization of the division. 



maintaining information on students and staff over an extended period 
of time, and must be capable of changing the content of data to be stored 
and analyzed as conditions warrant. 

The Division should work with teachers, students, administrators 
and others to develop means of collecting information from the field that 
will be useful immediately to teachers and others, and that will help 
def-ine ranges of and variations in instructional approaches and methods. 

Finally, the Division should be responsible for conducting 
research studies aimed at improvement of evaluation instrumentation, 
procedures, and utility, and of program or project selection, design and 
implementation. It should expand its Research Information Center; it 
should work closely with other divisions .departments and committees to 
develop studies to test new instrumentation (including educational achieve- 
ment tests) and to encourage use of unobtrusive measures of cognitive and 
social -emotional development, and to develop additional measures of system 
performance. 

There should be continuous and substantive involvement of repre- 
sentatives of different groups and agencies involved in the planning and 
design of research and evaluation studies. The Division must assure that 
there is clear agreement and understanding not only among operational and 
administrative groups, but also among community representati ves and others, 
about the roles and responsibilities of the Division's functions vis N a vis 
educational programs. 

C. Development of an Evaluation Data System 

An evaluation data system is described that will, over time, 
enable the school system to examine the effectiveness of different kinds 
of educational services and instructional programs with children of 



varying needs and backgrounds. The basic requirement of the system is 
to maintain a continuous record of students, and their history of enroll- 
ment, attendance and exposure to different instructional approaches and 
services. The system should also enable measures of achievement to be 
correlated with students to permit analyses of the comparative effec- 
tiveness of programs over time. 

The evaluation data system is not a general management information 
system or as a pupil, staff or program accounting system. In its initial 
development it would contain only data considered minimally necessary 
for purposes of evaluation of program effectiveness. The initial program 
identification data to be included emphasize reading instruction, since 
that is an area of particular concern in the schools, at this time. A 
preliminary version of a form for defining reading programs in terms of 
the approaches used in teaching reading at the classroom level has been 
developed and included in the report. In addition, identification of 
involvement of students and staff in some Federally funded programs not 
necessarily concerned with reading instruction will also be included. 

The plan proposed calls for pilot testi'ig of forms s procedures 
and data collection in sixteen elementary schools, two junior high schools 
and two senior high schools during the first year. Work on the auto- 
mated data system development (file design, programming etc.) would 
proceed concurrently. Selected reading programs at selected elementary 
grade levels would be evaluated as a means of demonstrating the uses of 
the evaluation system on a limited basis. Evaluations of programs at 
the secondary school level would not. be attempted during the first year. 
Effort at that level would be directed toward defining programs and 
data parameters that would be appropriate for evaluation purposes in 
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subsequent phases of system development and operation. 

At the end of the pilot phase, there will be some evaluation 
data, development of the computer support system, and information about 
the feasibility of extending the data system to include data on additional 
programs and services. 

The data system would be extended to additional elementary 
and secondary schools over a five year period unless early experience 
indicates that extension time can be compressed without risk, of overload 
or loss of quality of data. 

It is assumed that the Division of °lanning. Research and 
Evaluation will have prime responsibility for establishing the require- 
ments of the evaluation system, and that it will provide specifications and 
support to the Department of Automated Information Systems in developing 
computer based components of the system. The Division will also be 
responsible for manual collection of data from schools, teachers and 
other divi si Oils during the pilot phase. Finally, the Division will 
determine, in consultation with appropriate groups, the programs to be 
evaluated initially and the instruments to be used, and the testing 
schedules to be followed. 

A minimal evaluation can be made using tests administered early 

in the fall of each school year. The tests given, for example, in the 

third grade provide evaluative information about the programs given in 

the second grade the preceding year. An advantage of testing in the fall 

is that data can he returned to teachers in time to help, with their 

current classes. For some reading programs, there must be tests specific 

to the program to measure progress. However, it is assumed that at some 

point general achievement or performance tests will be employed, regard- 
less of specific reading program. 

ix 
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D. Special Innovative or Experimental Projects 

The Division of Planning, Research and Evaluation should plan 
to use the evaluation data system as a primary facility for assessing 
the long term effects of special projects. It should also undertake a 
series of analyses aimed at projecting the impact of special projects. 

To do this it must be able to obtain a number of elements of information 
from program and project directors elsewhere in the system, and to use 
this information to examine the relationship of projects to the overall 
objectives of the educational system. 

The Division should maintain an up-to-date file on all special 
projects and programs in the school system, and should provide technical 
assistance and guidelines to programs and projects in the planning and 
implementation of evaluations. 

Guidelines for analysis of the relationship of. the Title III 
program to the educational system are provided in the report. 

E. Further Research Studies 

The Division of Planning, Research and Evaluation should under- 
take several research tasks. One is to develop plans for obtaining 
measures related to system objectives that can not be derived from data 
available in the evaluation data system as currently conceived. A 
second is to assist in the development and testing of tests specifically 
designed for the city's population of students. A third is to develop 
unobtrusive or non-reactive measures of the cognitive, social and emotional 
development of students. A fourth is to explore varieties cf approaches 
to observing classrooms to maximize the value of such observations to' 
teachers in their regular instructional programs, as well as to continue 
to develop more specific information about the actual nature of instruc- 
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tional programs. Other tasks include development of further definitions 
and categorizations of instructional programs for inclusion in the eval- 
uation data base; and development of means of cross-referencing experi- 
mental, innovative or special project characteristics with ERIC codes 
and classifications. 
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FOREWORD 

This is the final report of a project undertaken with the D.C. 
Public Schools under Contract NS 7044. The primary objective was the 
design of a comprehensive evaluation system for the Title III Programs 
of the D.C. Public Schools. The specific tasks were to: 

1. design of a project data bank 

2. analyze program functions and identify methods 
of measuring them 

3. develop a preliminary system of categorizing 
objectives of educational programs. 

This report provides the conceptual design of a systemwide 
data base that incorporates Title III orojects. It provides recommen- 
dations for evaluating different functions and aspects of projects and 
of regular programs at different points in time. And it provides a 
preliminary categorization of objectives of the educational system to 
which programs may be related. 

In order to provide a focus for evaluation of innovative, 
experimental or special projects, as well as regular programs, the 
functions of a central unit for systemwide evaluation are discussed. 
Evaluation, as well as planning and research functions, are inter- 




related, and it is assumed that a single administrative unit could 

usefully provide support and services to the overall school system in 

* 

all three areas. The Division of Planning, Innovation and Research is 
assumed in this study to be that unit. 

The first four chapters of the report give brief statements 
about evaluation, problems, and strategies. Chapter V presents some 
suggested functions of a centralized planning, research and evaluation 
unit. Chapter VI presents suggested categories of system objectives, 
and some criteria of performance. Chapter VII discusses a general eval- 
uation data base and associated systems. Chapter VIII provides guide- 
lines and suggestions for evaluation of innovative projects, including 
Title III projects. Chapter IX lists briefly suggested research tasks. 
Chapter X provides a general development plan for the evaluation data 
system, geared initially to concentrate on reading programs. 

Appendices provide additional technical, support and background 
material, as well as a list of references found particularly useful for 
this study. 
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* The Division Is called the Division of Planning, Research 
and Evaluation throughout the report at the suggestion of the Acting 
Division Head. 
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I. INTRODUCTION 



This is a final report of a project to develop evaluation 
strategies for the D.C. Public School system. The original focus of the 
project was to develop a plan for the evaluation of the Title III pro- 
gram of the District of Columbia. The focus was broadened to the devel- 
opment of an overall strategy for the evaluation of educational programs 
of the entire school system. There is strong justification for such an 
extension, since there would be little rational basis for evaluating the 
Title III program (projects and administrative apparatus) in isolation 
from the needs and objectives of the education system of which the pro- 
gram is a component. To do so is to encourage precisely the kind of 
suosystem optimization (i.e., fractionation) that plagues educational 
planning and operations everywhere. The Title III program should have 
a coherent relationship to the total system. Furthermore, evaluations, 
not only of the Title III program, but of other special and regular 
programs in the system should have a coherent relationship to each other. 

Thus, at the request of the Acting Associate Superindendent 
for Planning, Innovation and Research, tre initial focus was broadened. 
This report describes a framework and recommended approaches for system- 
wide educational evaluation of regular programs and of special programs 
and projects, Including Title III projects. 
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II. SOME BASIC PREMISES 

The issue of evaluation is not whether it is done or not. The 
issue is how, by what rules of evidence , and according to whose criteria . 
That is, the fact that a program starts means t^at someone has made a 
projective evaluation. The fact that it continues means that some kind 
of process evaluation has been made. The fact that it is terminated, 
continued, extended or modified means that some kind of product evaluation 
has been made. 

The implications of these remarks are that there are different 
forms of evaluation, applicable at different times and to different ends 
or decisions. Evaluation requires design just as much as does instruc- 
tion. The design may be explicit or implicit, simple or complex, narrow 
or broad, pertinent or irre'.event, systematic or haphazard, public or 
private, brief or extended, or intermediate degrees of any or all of these 
and other dimensions. 

There are three basic premises underlying the strategies suggest- 
ed In this report. The first premise is that an evaluation function In 
the public school system entails a machinery, organization and set of 
processes that involves design, development, operations and assessment 
like any other najor educational function. 
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The second premise is that an evaluation system should be evo- 
lutionary both in the short run and in the long run. It can not meet 
all evaluation needs immediately , and it must be capable of changing 
(expanding, modifying, incorporating new data, etc.) after it is fully 
developed and implemented. Education itself is a continuous , evolving 
cyclical process. Evaluation, as a support function, should be similarly 
conceived. 

The third and last basic premise of the strategies proposed 
here is that evaluation is a function quite separate from program de- 
sign and decision-making. It is not up to the evaluator to decide what 
programs or approaches or curricula, etc., should be implemented. The 
^unctions of evaluation should be to help illuminate the nature of al- 
ternatives, to clarify what is happening or being done, and to provide 
information for decision-making about further courses of action. 
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III. SOME PROBLEMS WITH EVALUATION IN THE D.C. 

PUBLIC SCHOOL SYSTEM AT PRESENT 

The following is a list of what appear to be some of the reasons 
why evaluation of programs and projects in the public school system is 
for the most part limited in scope and utilization at present. These 
and other problems underlie the strategies proposed. 

1. Lack of regular budget funding to support: 

a. sufficient evaluation staff 

b. adequate machinery for handling evaluation data (col- 
lection, storage, retrieval, analysis, interpretation, etc.) 

c. systems for providing adequate and timely feedback of 
useful evaluation information to teachers, principals, 
supervisors, specialists, parents, community, admin- 
istration and other interested parties. 

2 . Lack of clear roles and responsibilities for evaluation in 
system programs and processes. 

3. Mobility of students. 

4. Multiplicity of programs and the technical and methodological 
problems posed by that condition. 
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5. Lack of an organizing .'ramework of objectives and goals for 
the D.C. Public School system; i.e., lack of focus stated 
in measurable terms. 

6. Lack of focus for evaluation {objectives, priorities,roles, 
etc. ) . 

7. Widespread suspicion, controversy and concern about the 
appropriateness, reliability, validity and use of tests 
and test scores. 

8. Legal and/or contractual restrictions on data collection. 

9. Negative attitudes and beliefs about evaluation - its roles, 
forms : time characteristics, strengths and limitations - and 
about factors affecting it, etc. 
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